Methods and system for occupancy class prediction and occlusion value determination

ABSTRACT

The present disclosure describes a method for occupancy class prediction, such as for occupancy class detection in a vehicle. In aspects, the method includes determining, for a plurality of points of time, measurement data related to an area and determining, for a plurality of points of time, occlusion values based on the measurement data. The method further includes selecting, for a present point of time, one of a plurality of modes for occupancy class prediction based on the occlusion values for at least one of the present point of time and a previous point of time and/or based on one of the plurality of modes for occupancy class prediction selected for the previous point of time. The method additionally includes determining, for the present point of time, one of a plurality of predetermined occupancy classes of the area based on the selected mode for the present point of time.

INCORPORATION BY REFERENCE

This application claims priority to European Patent Application NumberEP21188547.0, filed Jul. 29, 2021, the disclosure of which isincorporated by reference in its entirety.

BACKGROUND

Digital imaging devices, such as digital cameras, are used in automotiveapplications to monitor the interior of a vehicle. Interior sensingapplications are of great importance for the modern automotive industryand are expected to gain further popularity with increased focus ondriver assistance systems and self-driving cars. Cabin view cameras forexample may be able to replace traditional sensors (for example, for theseat belt alert function, air bag control or hands-off wheel detection)and may offer a wide range of additional possibilities.

However, it could frequently happen during normal driving scenarios,that the way between a region and a camera may be occluded even for alonger period of time. This may lead to inaccurate observing resultsfrom the camera or an observation of the region may be impossible atall. It is therefore desirable for the system to react to longer lastingocclusions and to make seat occupancy classification more reliable.

Accordingly, there is a need for methods and systems for occupancy classprediction and occlusion value determination that lead to reliable andaccurate results.

SUMMARY

The present disclosure provides a computer-implemented method, acomputer system, and a non-transitory computer readable medium accordingto the independent claims. Example embodiments are given in subclaims,the Description, and the Drawings.

In one aspect, the present disclosure may be directed at acomputer-implemented method for occupancy class prediction, such as foroccupancy class detection in a vehicle, wherein the method comprises thefollowing steps performed (in other words: carried out) by computerhardware components: determining for a plurality of points of timemeasurement data related to an area; determining for a plurality ofpoints of time occlusion values based on the measurement data; selectingfor a present point of time one of a plurality of modes for occupancyclass prediction based on the occlusion values for at least one of thepresent point of time and a previous point of time and/or based on oneof the plurality of modes for occupancy class prediction selected forthe previous point of time; and determining, for the present point oftime, one of a plurality of predetermined occupancy classes of the areabased on the selected mode for the present point of time.

In other words, based on the measurement data of an area, occlusionvalues of the area may be determined. Based on the occlusion values forat least one of the present points of time and a previous point of time,one of a plurality of modes for a present point of time may be selected.The one of a plurality of modes for a present point of time may also beselected based on the selected one of the plurality of modes for theprevious point of time or based on a combination of the occlusion valuesand the selected one of a plurality of modes for the previous point oftime. An occupancy class prediction of the area may be determined basedon the selected mode for the present point of time. For the occupancyclass prediction, a plurality of predetermined occupancy classes of thearea may be determined.

The area may be an environment around a sensor, like a camera, a radarsensor or a light detection and ranging (LIDAR) sensor. The area may beof variable size and/or the area may comprise the interior of a vehicle,a car, a truck or the like. The area may be a specific area (or sector)which may comprise a specific region of interest, for example a seatregion of a car (like a single seat or a seat bench).

The occlusion values may describe how relevant an occlusion may be todetermine an occupancy class of the area as described herein. Theocclusion may be a line-of-sight obstruction of the sensor, in otherwords the sensor may not be able to detect if or if not the area may beoccupied. Different kinds of obstacles may lead to the occlusion, forexample body parts of an occupant, objects or disturbances of the sensor(like contamination, fracture of a camera lens, dust, smoke or fog). Theoccupancy class may be a classification of different kinds of occupancyconditions. An occupancy condition may be, for example if a seat or aseat region of a passenger car is occupied, if a seat or a seat regionof a passenger car is not occupied. According to various embodiments,two or more occupancy conditions may be combined into a common class(e.g., a combined class may include an occupancy condition where a seator a seat region of a passenger car is not occupied (in other words: anempty seat) and an occupancy condition where the seat or the seat regionof the passenger car is occupied by an object).

The selected mode for the present point of time, which may be based onat least the occlusion values for at least one of the present points oftime and a previous point of time and which may be used for theoccupancy class prediction, may define a condition that specifies theocclusion. The selected mode may also determine (or set, or select, orindicate) an action for the occupancy class determination. There may bea plurality of modes, in particular a first mode, a second mode, a thirdmode and/or a fourth mode (which may also be referred to as last mode)as described herein.

The plurality of predetermined occupancy classes may describe differentoccupancy states, for example a child seat, a person, an object or anempty seat as described herein. One of the plurality of occupancyclasses or occupancy states may specify what kind of occupancy may bepresent in the area.

According to an embodiment, the plurality of modes comprises a firstmode, a second mode, a third mode and a last mode, wherein the firstmode is selected if the area is fully visible or slightly occluded, thesecond mode is selected if there is a short-time critical occlusion inthe area, the third mode is selected if there is a longer lastingcritical occlusion in the area, and the last mode is selected if thereis a longer lasting critical occlusion in the area and a stable state ofone of the plurality of predetermined occupancy classes is not reached.

According to an embodiment, in the first mode a change from a occupancyclass for the present point of time to a different occupancy class ofone of the plurality of predetermined occupancy classes may be based ona time fused occupancy class prediction.

The first mode may be a current mode for the occupancy class predictionif the area is fully visible or only slightly occluded. The area may befully visible if there is no occlusion between the camera and the area,i.e., the camera may be able to capture the total area without anyocclusion. The area may be slightly occluded if there are only smallparts or a small percentage of the area occluded, for example less than5%, 10%, or up to 20% of the total area. The time fused occupancy classprediction may be a combination of the occupancy class and a previouslyidentified occupancy class. The presently identified occupancy class maybe determined or predicted at a present point of time and the previouslyidentified occupancy class may be determined or predicted at a previouspoint of time. The previous point of time may be immediately before thepresent point of time (in other words: no further point of time isbetween the previous point of time and the present point of time) but isnot restricted to this. It will be understood, that a discrete sequenceof points in time may be used, for example equidistant points in time,for example a point of time every pre-determined amount of seconds, forexample every second, or every 1/10 of a second (i.e., 100 ms) or thelike. The presently identified occupancy class and the previouslyidentified occupancy class may be combined by a fusion parameter. Thefusion parameter may determine an impact of the presently identifiedoccupancy class and the previously identified occupancy class to thetime fused occupancy class prediction.

According to an embodiment, the selection of one of the plurality ofmodes may be provided from the first mode into the second mode, if theocclusion value for the present point of time is larger than apredetermined occlusion threshold, or from the second mode into thefirst mode, if the occlusion value for the present point of time islower than the predetermined occlusion threshold; or from the secondmode into the third mode, if the second mode has been selected for aperiod of time longer than a predetermined blocking time threshold; orfrom the third mode into the first mode, if a time fused occlusion valueis lower than the predetermined occlusion threshold; or from the thirdmode into the last mode, if a stable state of one of the plurality ofoccupancy classes is not reached for a predetermined occlusion timethreshold, no occupancy class reaches an increased confirmationthreshold for the predetermined occlusion time threshold, or theoccupancy class may constantly change for the predetermined occlusiontime threshold; or from the last mode into the third mode, if a stablestate of one of the plurality of occupancy classes is reached or a usermay confirm a change of the presently identified occupancy class; orfrom the last mode into the first mode, if the time fused occlusionvalue is lower than the predetermined occlusion threshold. A stablestate of one of the plurality of occupancy classes may be reached ifthere is no occupancy state change for a time period of a pre-determinedlength, e.g., 30 s up to 5 min. The pre-determined length may be theocclusion time threshold.

The selection of the modes may be a switch of the modes or a change ofthe modes, wherein the switch or change may be dependent on theocclusion value and/or time parameters. The time parameters, for examplea blocking time interval or a predetermined blocking time threshold, maybe a period of time that indicates how long a current mode may be validuntil the switch to another mode will occur. The predetermined occlusionthreshold may determine an occlusion value that may describe avisibility of the area, which may be necessary to predict the occupancyclass in the first mode. The predetermined occlusion threshold may bedetermined statistically such that the performance for occluded examplesbelow the predetermined occlusion threshold may not be significantlyworse than for not occluded cases of the method described herein. Thepredetermined occlusion threshold may be expressed as a numeric value,wherein the numeric value may describe how much of the area may beoccluded and the numeric value may be dependent on how well the methoddescribed herein may perform on occluded cases and on the position ofthe occlusion in the area. The larger the occlusion value, the lower maybe the visibility of the area. In the first mode the occupancy classprediction may be based on the time fused occupancy class prediction asdescribed herein.

The second mode may be a mode where short timed critical occlusionscenarios may be handled in such a way that no time fused occupancyclass prediction may be determined and no occupancy state changes maytake place. The second mode may be the current mode for only a shorttime instance, i.e., the blocking time interval may be a short timeinstance, e.g., a few seconds, until a mode change or switch of the modemay occur. The time instance or the blocking time interval of the secondmode may be determined by the predetermined blocking time threshold, inother words the predetermined blocking time threshold may define howlong the occupancy class prediction may stay in the second mode.

In the third mode, which follows the second mode if the blocking timeinterval is larger than the predetermined blocking time threshold, apredetermined occupancy class “object” and a predetermined occupancyclass “empty seat” may be combined to one class. In the third mode theoccupancy class prediction may try to reach a stable occupancy state ofone of the plurality of occupancy classes despite a longer lastingcritical occlusion. The stable occupancy state may be a state where anoccupancy class may be clearly determined by the method describedherein.

The last mode may be the current mode, if the method described herein isnot able to determine a stable occupancy state of the predeterminedoccupancy class in a predetermined occlusion time threshold. Thepredetermined occlusion time threshold may be a configurable timeinterval or a continuing time period to control how fast the system mayneed to react in occluded cases, e.g., 30 s or up 5 min. Thepredetermined occlusion time threshold may be dependent on differentdriving scenarios (for example, a speed or steering angle, or whether alane change is currently taking place, or whether the vehicle is drivingautonomously or not) and/or on a confidence value before the occlusion,for example, a high confidence value may lead to a larger predeterminedocclusion time threshold. Also, the last mode may be the current mode,if no occupancy class reaches an increased confirmation threshold forthe predetermined occlusion time threshold. The increased confirmationthreshold may be a predetermined value of one of the plurality ofoccupancy classes that indicate a reliable determination of therespective occupancy class. In the last mode there may be help required,for example from the driver of the passenger car. The driver may beasked to remove the occlusion or the driver may have to verify anoccupancy state change. The occupancy state change may be a change or aswitch of a current occupancy class of one of the plurality of occupancyclasses to a different from the current occupancy class of one of theplurality of occupancy classes.

According to an embodiment, the predetermined occlusion threshold maydepend on the occupancy class for the present point of time.

According to an embodiment, the method further comprises the followingstep carried out by the computer hardware components: determining acondition of the vehicle, wherein selecting one of the plurality ofmodes is based on the condition of the vehicle. The condition of thevehicle may describe a driving scenario, for example, the vehicle isdriving, the vehicle drives through a curve or on a highway, the vehicleis standing at a traffic light or the condition of the vehicle maydescribe a beneficial situation, for example the driver is not busy orfocused on the traffic.

According to an embodiment, the occlusion value may be determined basedon the measurement data and characteristics related to a relevantregion. The measurement data may be determined based on data from acamera system. The camera system may comprise one camera or a pluralityof cameras. Measurement data recorded by the camera system may be usedto detect RGB (red green blue) information, e.g., identify objects,persons, child seats, an interior of a vehicle, in particular seatregions of a vehicle, or infrared (IR) images which may be lesssensitive to varying lighting conditions (for example, at night time)and the measurement data may have high resolution, e.g. 1920×1080pixels. The camera system may be mounted in the interior of a passengercar and directed to the rear end of the passenger car. Thecharacteristics to the relevant region may describe properties of theregion that specify an influence of an occlusion in the region to theoccupancy class prediction. Further, the characteristics may bedetermined for each pixel of a plurality of pixels of the region. Therelevant region may be a part of the area of the measurement data or maybe the area of the measurement data.

According to an embodiment, the method further comprises the followingstep carried out by the computer hardware components: determiningcharacteristics related to a relevant region for the occupancy class forthe present point of time of the area from a plurality ofcharacteristics for each possible predetermined occupancy class of aplurality of predetermined occupancy classes; and wherein the occlusionvalue is determined based on the measurement data and the determinedcharacteristics related to the relevant region.

In another aspect, the present disclosure may be directed at acomputer-implemented method for occlusion value determination, themethod comprises the following steps performed (in other words: carriedout) by computer hardware components: determining for a plurality ofpoints of time measurement data related to an area; determining for aplurality of points of time characteristics related to a relevant regionfor a occupancy class for a present point of time of the area from aplurality of characteristics for each possible predetermined occupancyclass of a plurality of predetermined occupancy classes; and determiningfor the present point of time an occlusion value based on themeasurement data for a present point of time and the determinedcharacteristics for a present point of time related to the relevantregion. The plurality of characteristics for each possible predeterminedoccupancy class of a plurality of predetermined occupancy classes classmay be determined in advance for each of the predetermined occupancyclasses.

According to an embodiment, the characteristics related to the relevantregion may depend on the presently identified occupancy class for apresent point of time. The characteristics related to the relevantregion may comprise a weight mask, the weight mask may comprise aplurality of mask pixels, each mask pixel may correspond to ameasurement pixel of the measurement data. The weight mask andrespectively each of the plurality of mask pixels may determine theimpact of each of the plurality of measurement pixels to the occupancyclass prediction. Each of the mask pixels may be represented by anormalized scalar value, e.g., between 0 and 1. The larger the scalarvalue, the larger may be the impact of the mask pixel to the occupancyclass prediction and vice versa.

According to an embodiment, an occlusion prediction may be determinedbased on the measurement data, a pixel segmentation or an objectclassification. The occlusion prediction may determine an occludedregion of the measurement data. In other words, the occlusion predictionmay estimate a region or a plurality of regions of the measurement data,where the area of the measurement data is occluded in that the area ofthe measurement data may not be visible. There may be different methodsfor determining the occlusion prediction. The methods may be divided indirect methods and indirect methods. The direct methods may determinethe occlusion prediction by a pixel wise segmentation of the occludedregion. In other words, each pixel of the occluded region may becharacterized as occluded. The sum of all occluded pixels may define theocclusion prediction of the region. The indirect methods may be based onother computation modules, for example object detection algorithms orbody part tracking methods. The occlusion prediction based on indirectmethods may comprise all parts of the result of the object detectionalgorithm or the result of the body part tracking methods.

According to an embodiment, the predetermined occupancy classes maycomprise at least two of the following: an empty seat, an object, achild seat, and a person. A combination of two of the predeterminedoccupancy classes may be possible, for example, the occupancy classempty seat E and the occupancy class object O may be combined to oneclass.

According to an embodiment, the area may comprise a seat region. Theseat region may be a single seat of a passenger car, e.g., the middlerear seat of a car, the driver's seat, the co-driver's seat of a car, orthe rear seat bench of a car.

In another aspect, the present disclosure may be directed at a computersystem. The computer system may comprise a plurality of computerhardware components configured to carry out several or all steps of thecomputer-implemented methods described herein.

The computer system may comprise a plurality of computer hardwarecomponents (for example, a processor, for example processing unit orprocessing network, at least one memory, for example memory unit ormemory network, and at least one non-transitory data storage). It willbe understood that further computer hardware components may be providedand used for carrying out steps of the computer-implemented methods inthe computer system. The non-transitory data storage and/or the memoryunit may comprise a computer program for instructing the computer toperform several or all steps or aspects of the computer-implementedmethods described herein, for example using the processing unit and theat least one memory unit.

In another aspect, the present disclosure may be directed to a vehicle,comprising the computer system described herein and a sensor, whereinthe measurement data may be determined based on an output of the sensor.The sensor may be a camera.

In another aspect, the present disclosure is directed at anon-transitory computer readable medium comprising instructions forcarrying out several or all steps or aspects of the computer-implementedmethods described herein. The computer readable medium may be configuredas: an optical medium, such as a compact disc (CD) or a digitalversatile disk (DVD); a magnetic medium, such as a hard disk drive(HDD); a solid state drive (SSD); a read only memory (ROM), such as aflash memory; or the like. Furthermore, the computer readable medium maybe configured as a data storage that is accessible via a dataconnection, such as an internet connection. The computer readable mediummay, for example, be an online data repository or a cloud storage.

The present disclosure is also directed at a computer program forinstructing a computer to perform several or all steps or aspects of thecomputer-implemented methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments and functions of the present disclosure aredescribed herein in conjunction with the following drawings, showingschematically:

FIG. 1 illustrates characteristics related to a relevant region forpredetermined occupancy classes: child seat, empty seat, object andperson;

FIG. 2 illustrates relevant region and corresponding characteristicsrelated to the relevant region;

FIG. 3 illustrates weight masks for predetermined occupancy classes:child seat, empty seat, object, and person;

FIG. 4 illustrates occlusion examples of a camera view on a given seat;

FIG. 5 illustrates occlusion values for an example occlusion predictionand different predetermined occupancy classes: child seat, empty seat,object, and person;

FIG. 6 illustrates a flow diagram illustrating the relationship betweenthe modes for an occupancy class prediction based on the occlusionvalue;

FIG. 7 illustrates a flow diagram illustrating a method for occupancyclass prediction according to various embodiments;

FIG. 8 illustrates a flow diagram illustrating a method for occlusionvalue determination according to various embodiments; and

FIG. 9 illustrates a computer system with a plurality of computerhardware components configured to carry out steps of acomputer-implemented methods for occupancy class prediction and forocclusion value determination according to various embodiments.

DETAILED DESCRIPTION

The present disclosure may be directed to a process chain for handlingoccluded seat regions for occupancy class prediction, such as foroccupancy class detection in a vehicle. Implementations of the presentinvention and its operation are hereinafter described in detail for therear middle seat of a passenger car, which may be mainly affected byocclusion, but may be applied to other seats or seat regions and/orother vehicles like trucks.

Cabin view cameras for example may be able to replace traditionalsensors and may offer a wide range of additional possibilities, forexample improved airbag control by detecting child seats, reliablydistinguishing persons from heavy objects, driver distraction detection,driver drowsiness detection, detecting forgotten objects in the car orgesture-controlled applications.

An essential foundation for most applications may be a reliable seatoccupancy classifier which may assign a seat occupancy state for eachseat. Seat occupancy classification systems may use common methods forobject detection or person detection: image classification; objectdetection approaches; body keypoint detection; semantic segmentation;and/or a combination of several approaches to combine some advantages.As the seat occupancy state may be used for safety criticalapplications, the tolerance for classification errors should be close tozero.

If the camera is placed in the front (rear facing), especially the rearseats may often be occluded by persons or objects from the front seatsor adjacent rear seats. Occlusion problems may also occur for othercamera positions in a similar way. Occlusions may likely reduce thesystem performance, for example either because the visibility of theactual seat occupancy class may be reduced or because the occludedobject may be confused with another (e.g., wrong) seat occupancy class.The risk of misclassifications may depend on how much is occluded andwhether the actual seat occupancy class may be still recognizable. Apossible solution may be to treat occluded frames as blocked, forexample, if occlusion is detected in the running system the respectiveframes may be ignored and the seat occupancy state may not be updated.Even though this may prevent occlusion related classifier errors, it maycause the system to be blocked for a very long time and miss significantstate changes.

In order to make use of partially occluded areas while minimizing therisk of occlusion related classification errors, a cascade structurewith for example four different modes may be provided. The first mode(which may be named as regular mode 602) may deal with fully visible andslightly occluded areas only. Short timed critical occlusion scenariosmay be handled in the second mode (which may be named as short-termocclusion mode 604). In the third mode (which may be named as long-termocclusion mode 606), the system may try to reach a stable state of oneof a plurality of occupancy classes despite longer lasting criticalocclusion. The last mode (which may be named as Human Machine InterfaceHMI occlusion mode) may include a module which may seek assistance fromthe passengers if the system is unable to reach a stable state duringlonger lasting occlusion scenarios. Changes between the modes may dependon how much of the area is occluded, how long the occlusion may bepresent and the reliability of the predicted occupancy class.

Essential for handling occlusion may be to know the impact fromdifferent parts of the seat to the occupancy class determination. Forthis purpose, a weight mask may be generated as part of a trainingprocess for a plurality of predetermined occupancy classes, where theweight mask may include a plurality of mask pixels and each mask pixelmay correspond to a measurement pixel of the measurement data. Each ofthe mask pixels may be weighted based on the relevance for the occupancyclass prediction. The mask pixels may define characteristics related toa relevant region of the measurement data. The weight masks may becreated separately for each possible predetermined occupancy class ofthe plurality of predetermined occupancy classes, which may allowoccupancy class dependent differences. For example, persons or childseats may be reliably classified if a lower part of the middle seat oranother seat region of a passenger car is occluded, whereas fordetecting an empty seat of a passenger car, it may be essential to havea visible seat region of passenger car. For creating the pixel weightsof the weight masks, a classifier trained on clean data withoutocclusion may be used.

FIG. 1 shows characteristics related to a relevant region 102, 104, 106,108 for predetermined occupancy classes respectively, visualized on topof an empty seat. The predetermined occupancy classes may be for examplea child seat C, an empty seat E, an object O, and a person P. A cameramay be placed in the front (rear facing) of the passenger car to monitorespecially the rear seats that may often be occluded by persons P orobjects O from the front seats or adjacent rear seats (as shown in FIG.4 ). The relevant region 102, 104, 106, 108 may depend on thepredetermined occupant class. In FIG. 1 the relevant region 102 for achild seat C, the relevant region 104 for an empty seat E, the relevantregion 106 for an object O and the relevant region 108 for a person Pare identified by a dotted line, respectively. FIG. 1 shows an examplehow the weight masks may look like. In this example, the Grad-CAM(gradient-weighted class activation mapping) method was used tohighlight image parts where the classifier is looking at in order todecide for a predetermined occupancy class. A combined weight map may becreated by calculating the average over all training examples for eachpredetermined occupancy class.

FIG. 2 shows a relevant region on the left side of FIG. 2 andcorresponding characteristics 204 related to the relevant region on theright side of FIG. 2 . The relevant region is identified by a frame 202.The corresponding characteristics 204 related to the relevant region onthe right side may be scaled for visualization. Characteristics 204 orweights may only be estimated or calculated for the relevant region.Other parts of the area 206, exclusive the relevant region, may not belabeled by characteristics 204 or weights and are represented in blackcolor in FIG. 2 .

Alternatively, the weight mask and thus the characteristics 204 of oneof a plurality of predetermined occupancy classes may be created bytraining a classifier on clean data and evaluating the classifierperformance on occluded examples. In this case, higher weights indicateareas where occlusion statistically may lead to a reduced classifierperformance. The weight mask may be normalized so that the sum of allpixels in the weight mask is equal to 1.

Occlusion may be simulated by blackening random parts of the imagesbefore running the classifier. The reduction of the occupancyclassification result compared to the unoccluded (in other words: freeof an occlusion) case may be used as measurement for the impact of theocclusion. All occluded pixels of this image may be weighted with thecalculated value of the procedure described before. After doing this foreach example, the weight mask may be created by calculating the meanover all examples and then normalizing over the relevant region. FIG. 3again shows weight masks for the predetermined occupancy classes childseat C, empty seat E, object O, and person P. The intensity of theweight mask determines the impact of the occlusion for the occupancyclass prediction. The higher the intensity (i.e., higher weights), thelower the classifier performance if an occlusion is determined in theregion with high intensity of the weight mask. A region of intensity302, 304, 306, 308 is shown in FIG. 3 for each of the predeterminedoccupancy classes child seat C, empty seat E, object O and person P. Theregion of intensity 306 of an object O may represent for example asmaller region compared to the regions of intensity 302 of a child seatC, or compared to the regions of intensity 308 of a person P, since asmall object O may be harder to identify than a child seat C or a personP. The region of intensity 304 of an empty seat E may be the smallestregion compared to the region of intensity 302 of a child seat C, theregion of intensity 306 of an object O or the region of intensity 308 ofa person P, since it has to be secured that the seat is really notoccupied also by very small parts like mobile phones or the like.

There may be various ways how the camera view on a given seat may beoccluded. FIG. 4 shows occlusion examples of a camera view on a givenseat. For example, a middle-seat may be occluded based on a head of adriver 402, or may be occluded based on an arm of a driver 404, or otherbody-parts of a driver (not shown in FIG. 4 ). Also, an occlusion basedon an arm of a co-driver 406 and/or other body-parts of a co-driver (notshown in FIG. 4 ) may occur. Furthermore, an occlusion based on anobject 408, for example a mobile phone or the like or an occlusion basedon an object 408 and an occlusion based on body-parts of passengers (forexample, a driver, a co-driver, or passengers on the rear seats) maylead to disturbance when determining an occupancy class. Furthermore, anocclusion based on body-parts like an arm of a rear-seat passenger 410or an occlusion based on body-parts of a rear-seat passenger or based onan object 412 in the line-of-sight of the camera may influence theoccupancy class prediction.

There may be several methods to detect and predict occlusions. There maybe provided direct methods to estimate occlusions as well as indirectmethod based on other computation modules, for example object detectionor body part tracking. Also, a combination of direct methods andindirect methods for occlusion prediction may be possible.

A pixel wise segmentation of occluded seat regions may be an example fora direct method to estimate occlusions. A segmentation module may beused to provide a pixel wise occlusion mask specifying occluded areas.The segmentation module may contain a segmentation network that may betrained on an annotated set of occluded images.

Parts of the seat may be marked as occluded if positively classifiedobstacles in line-of-sight, for example objects O or persons P,associated to another seat may partly cover the observed seat region.For example, if body keypoints from another seat are covering theobserved seat region, the occluded region may be determined based on askeleton model and expected aspect ratios for each body part. Regions inthe image next to the body key points/joints or connecting lines of theskeleton model may be masked out as occluded. The influence area of themasking may be controlled via parameters, for example, one could use athinner or thicker line to connect the body joints. The masking mayfurther be weighted, e.g., with fading support the further away from theskeleton. Another example may be, if bounding boxes from another seatare covering the observed seat region, all pixels inside the boundingbox may be marked as occluded. Also, object segmentation masks may beused to detect and predict occlusions. The pixels inside objectsegmentation masks from other seats may be marked as occluded.

After an occlusion prediction 502 (as shown in FIG. 5 ) has been carriedout, an occlusion value OCC may be determined based on the measurementdata. The occlusion value OCC may be a scalar value and may estimate theocclusion impact to the occupancy class prediction. The occlusion valueOCC may be calculated by:

$\begin{matrix}{{OCC} = \frac{\sum\limits_{i = 0}^{m - 1}{\sum\limits_{j = 0}^{n - 1}{{occlusion\_ map}{\left( {i,j} \right) \cdot {{heatmap}\left( {i,j} \right)}}}}}{\sum\limits_{{i = 0},}^{m - 1}{\sum\limits_{j = 0}^{n - 1}{{heatmap}\left( {i,j} \right)}}}} & (1)\end{matrix}$where i and j may denote pixel-coordinates of measurement data (like animage) with size i×j captured by the sensor.

The term occlusion_map(i,j) may be referred to as the occlusionprediction 502 and heatmap(i,j) may describe the characteristics of therelevant region. Therefore, the occlusion value OCC for one of aplurality of occupancy classes may be determined by a pixel wisemultiplication of the occlusion prediction 502 (in other words:occlusion map) and the mask pixels of the characteristics (in otherwords: heatmap) for a plurality of occupancy classes and normalizationto the mask pixels of the characteristics for a plurality of occupancyclasses. The occlusion value OCC may be proportional to the amount ofocclusion of the seat region, where each pixel on the seat area may beweighted based on the predefined characteristics related to the relevantregion in order to consider that occlusions of highly important imageregions for the classification may be more critical than regions thatmay have almost no influence on the occupancy class prediction. Thecharacteristics related to the relevant region may be obtained from theprevious seat occupancy class.

FIG. 5 shows an example of occlusion value determination for anocclusion prediction 502, for example an arm of a driver, and thedifferent predetermined occupancy classes: child seat C, empty seat E,object O, and person P. The occlusion value OCC for the predeterminedoccupancy class empty seat E may be estimated based on the equation 1herein, viz. a pixel wise multiplication of the occlusion prediction 502with the characteristics 506 related to an empty seat E and normalizedto the characteristics 506 related to an empty seat E. The occlusionvalue OCC for the predetermined occupancy class object O may beestimated based on the equation 1 herein, viz. a pixel wisemultiplication of the occlusion prediction 502 with the characteristics508 related to an object O and normalized to the characteristics 508related to an object O. The occlusion value OCC for the predeterminedoccupancy class child seat C may be estimated based on the equation 1herein, viz. a pixel wise multiplication of the occlusion prediction 502with the characteristics 504 related to a child seat C and normalized tothe characteristics 504 related to child seat C. The occlusion value OCCfor the predetermined occupancy class person P may be estimated based onthe equation 1 herein, viz. a pixel wise multiplication of the occlusionprediction 502 with the characteristics 510 related to person P andnormalized to the characteristics 510 related to a person P. Theocclusion value OCC determination for the predetermined class empty seatE results in an occlusion value OCC of 0.82 for this example, since theocclusion prediction 502 overlaps a big area of the characteristics 506related to an empty seat E. On the other side, the occlusion valuedetermination for the predetermined class person P results in acomparatively low occlusion value OCC of 0.38 for this example, sincethe occlusion prediction 502 overlaps a small area of thecharacteristics 510 related to a person P.

The determined occlusion value OCC may then be used to determine a classdependent predetermined occlusion threshold TC, which may statisticallyindicate a significantly increased risk for classification errors.Considering the predetermined occupancy class may allow an occupancyclass dependent behavior, for example that persons P or child seats Cmay often be still visible and may be correctly classified despite highocclusion value OCC, whereas empty seats E or objects O with occlusionmay easily be misclassified, even if only a small part of the area maybe occluded.

Based on the occlusion value OCC, a mode for occupancy class predictionmay be determined. There may be a plurality of different modes like aregular mode 602, a short-term occlusion mode 604, a long-term occlusionmode 606 and/or an HMI occlusion mode 608. In other words, there may bea state machine which may include four states where each state mayrepresent an operation mode. The transitions between the states maydepend on the occlusion value OCC, and/or time parameters and/or (incase of 608) a reliability of the predicted occupancy class result. FIG.6 shows a flow diagram illustrating the relationship between the modesfor an occupancy class prediction based on the occlusion value OCC.

The regular mode 602 may be the default mode for fully visible andslightly occluded frames of the measurement data. In this mode, statechanges between for example the four occupancy classes empty seat E,object O, child seat C and person P may be done based on the time fusedseat occupancy class predictions.

Another mode may be the short-term occlusion mode 604. The systemswitches from the regular mode 602 to short-term occlusion mode 604 oncea visibility value, in particular the occlusion value OCC, exceeds acritical predetermined occlusion threshold TC. If the occlusion valueOCC drops below the predetermined occlusion threshold TC, the systemswitches back to regular mode 602. In short-term occlusion mode 604,frames of the measurement data may be ignored completely and theprocessing results may not be considered in the time fusion at all.State changes of the seat occupancy classes may not be possible in theshort-term occlusion mode 604. The short-term occlusion mode 604 may beactive only for a predetermined period of the time the short-termocclusion mode 604 has been selected. This predetermined period of timemay also be called as a blocking time interval TB and may be only ashort-time interval, e.g., a few seconds. The blocking time interval TBmay be parameterized and adjusted based on a given application toreflect the acceptable delay of a status update. If the blocking timeinterval TB is exceeded, the system switches to the long-term occlusionmode 606.

In the long-term occlusion mode 606, the processing results may be usedagain in a fusion step to update the output, i.e., the occupancy stateprediction over time. However, the approach may be changed compared tothe regular mode 602. For example, requirements for state changes may beincreased by raising the corresponding thresholds for the time fusedoccupancy class prediction. The probability for the predicted occupancyclass being correct may strongly exceed the probability forocclusion-based classification errors. Furthermore, in the long-termocclusion mode 606, the class empty seat E and the class object O may becombined to one class in order to ensure the classifier performance forthe more essential classes person P and child seat C. For switching backto regular mode 602, a time fused occlusion value TOCC may becalculated. This may be mainly important to prevent immediate seatoccupancy class changes in regular mode 602 after single frames of themeasurement data with high visibility. In another embodiment, ahysteresis may be applied for the time fused occlusion value TOCC. Thesystem may switch to the HMI occlusion mode 608 if no clear output,i.e., an unstable occupancy class, is reached. The unstable occupancyclass may be defined in that several occupancy classes are equallylikely, and/or no occupancy class reaches the increased confirmationthreshold in the long-term occlusion mode 606, or the output result,i.e., the occupancy class prediction may be constantly changing withinconsecutive frames of the measurement data for a configurable timeinterval.

The HMI occlusion mode 608 may be processed if the system is unable toreach a stable occupancy class prediction for a longer time period,e.g., 30 s up to 5 min. The HMI occlusion mode 608 may be terminated andthe system may switch back to the long-term occlusion mode 606 if astate change has been confirmed by the driver via a human machineinterface or a stable state of one of the plurality of occupancy classesis reached. A stable state of one of the plurality of occupancy classesmay be reached if there is no occupancy state change for a time periodof a pre-determined length (for example, as indicated by the occlusiontime threshold), e.g., 30 s up to 5 min. In the HMI occlusion mode 608,the human machine interface may communicate with the driver to eithereliminate the source of the occlusion or give feedback on the currentoccupancy state, for example a current seat occupancy state. Thisinterface may be intended to provide the following functions: furtherspecify the type of occlusion (for example, which parts are occluded,the direction the occlusion comes from or whether it comes from anobject or body parts) by an additional module and may use the result torequest the driver to remove the source of occlusion; the driver may beasked to verify the new occupancy class before an occupancy class change(in other words: occupancy state change). The system may be able toprovide various signals that may be used to steer the dialog with thedriver and may enable to make the dialog interactive, for example, incombination with a smart assistant, including information what may beblocking the view (body part, object, something close to the camera, orlighting), information what may be the expected occupancy class, thatcannot be confirmed due to the occlusion.

Examples for dialogue elements may include, “[w]elcome on board. I amhaving difficulty seeing if everyone is safely seated in the rear seats.Could you please assist me by”, in one example, “moving your arm,shoulder, upper body to the side,” “confirming there is a child seat(C), person (P), object (O) on the middle rear seat,” and/or “unblockingmy view.” The user may then confirm the request via a voice command,body movements, and/or hand gestures.

The mode change or switch of the mode may be dependent on a vehiclecondition. For example, the mode change or switch of the mode may onlyoccur if the vehicle is standing, e. g. at a traffic light or stop sign,if the vehicle is driving or if the driver of the vehicle will not getdistracted by the mode change or switch of the modes, e.g., duringautonomous or semi-autonomous driving modes.

Especially, before switching to the HMI occlusion mode 608, it may beensured that the interface can be applied safely without distracting thedriver. This may for example be the case while the car is standing.Depending on the automation level of the car, it may further be appliedin circumstances where the driver doesn't need to take other actions atthe same time.

To increase the stability of the seat occupancy class prediction, theframe-based result may be stabilized over time, for example by atemporal filtering:class_pred_fused=a*class_pred+(1−a)*class_pred_fused  (2)where equation 2 may be a formula for an exponential fusion and “a” maybe a constant fusion parameter.

In a fusion step, the occlusion value OCC may be used while fusing theseat occupancy class prediction over time to reduce the influence ofoccluded frames of the measurement data. Frames of the measurement datawith larger occlusion values OCC may have smaller influence on the fusedresult:class_pred_fused=b*class_pred+(1−b)*class_pred_fused  (3)where b=a*(1−occlusion_value) may be a fusion parameter.

With this choice of fusion parameter, so for larger occlusion values,the class prediction from the current frame may have a smaller influenceon the fused class prediction.

One problem for occupancy class prediction may be a decision between anobject O and an empty seat E for occluded examples. The decision betweena small object O and an empty seat E may be a hard case even in normalcircumstances and may become especially difficult for occluded examples.Small objects O, like keys or mobile phones, may get occluded much moreeasily than persons P or child seats C. Similarly, even small occludedparts on an empty seat E may easily be mistaken as an object O. Hence,in long-term occlusion mode 606 the decision between an object O and anempty seat E may be omitted, in order to ensure the required performanceon the other occupancy classes person P and child seat C, which may beconsidered more important.

For occluded frames of the measurement data, the fused occupancy classprediction for object O and empty seat E may be still calculatedseparately. However, in the state machine used to control the systemstate, the two occupancy classes object O and empty seat E may beconsidered as one class, with the classification result being the sum ofboth occupancy class predictions. In case of a state switch to thisclass, the system output, i.e., the occupancy class prediction, mayindicate that both occupancy classes (object O or empty seat E) may bepossible. If the previous occupancy class is object O or empty seat E,the occupancy class may be kept, if the corresponding occupancy classprediction may be higher than each of the others.

The HMI occlusion mode 608 may provide an interface to get assistancefrom the driver in case the system may be unable to provide a reliableprediction for the seat occupancy class. The following modules may beapplied one after the other.

First, advices may be given to the driver on how to remove the source ofocclusion. The goal of this module may be the removal of the occlusionby the driver based on a system indication. The driver should receiveconcrete information on what kind of occlusion needs to be removed. Themodule may need further steps to specify the kind of occlusion.Following functions may be provided, but may be replaced by differentapproaches.

If the occlusion was detected indirectly by body key points or an objectdetection output for other seats, the available information about thekind of occlusion may be used.

An additional classifier may be trained on a bunch of moderately orstrongly occluded examples. The output of the additional classifier mayspecify from which of the other seats the occlusion comes and/or whetherthe occlusion may come from a body part or an object. The output of theadditional classifier should include a confidence value in order toavoid giving incorrect indications.

Occlusion masks may be used to further specify the occluded area and/orto check whether the occluded area may be limited to specific parts ofthe seat, for example a seating area or backrest.

After the advices to the driver on how to remove the source ofocclusion, a confirmation from the driver for a state change may berequested. The driver may be asked to verify the new state, i.e., thenew occupancy class, before the state change will be carried out. Thefused occupancy class prediction of a class confirmed by the driver mayneed to be increased by an additional confidence value to stabilize theresult for future state changes.

The following example may show how occluded data with reduced influenceon the occupancy class prediction may be added to a training set for aseat occupancy classifier with lower risk of reducing the performance.Examples with larger occlusion values OCC may have lower influence onthe training compared to fully visible examples, consequently thefailure tolerance for the seat occupancy classifier (during training)may be higher for larger occlusion values OCC. The occlusion value OCCmay be calculated for each example in the training set based on amanually labeled occlusion mask and the characteristics to a relevantregion (in other words: importance heatmaps) for the ground truth seatoccupancy class.

By considering the occlusion value OCC during training, frames of themeasurement data which are expected to have a lower influence on theoccupancy class prediction may be already considered less important intraining.

During classifier training, the occlusion value OCC may be used in aloss function to reduce the influence of the seat occupancy classprediction from occluded examples to the training loss. As an example,the following combined loss function for occlusion and seat occupancyclassification task, may be defined:

Classification loss: crossentropy (pred_class);

Occlusion loss: crossentropy (pred_occlusion); and

Total loss: Occlusion loss+(1−occlusion_value)*classification loss.

The decision between object O and empty seat E during a convolutionalneural network (CNN) training may be treated as follows: In case theground truth occlusion is larger than the allowed threshold for regularmode 602, empty seat E and object O may be treated as the same classwhile calculating the classification loss. The values for both classesmay be added to get the merged ground truth and prediction values.

FIG. 7 shows a flow diagram 700 illustrating a method for occupancyclass prediction according to various embodiments. At 702, measurementdata related to an area may be determined. At 704, occlusion valuesbased on the measurement data related to the area may be determined. At706, a plurality of modes for occupancy class prediction for a presentpoint of time may be selected based on the occlusion value OCC and/orbased on one of the plurality of modes for the previous point of time.At 708, one of a plurality of predetermined occupancy classes for thepresent point of time of the area may be determined based on theselected mode for the present point of time.

According to an embodiment, the plurality of modes comprises a firstmode (for example, 602), a second mode (for example, 604), a third mode(for example, 606) and a last mode (for example, 608), wherein the firstmode is selected if the area is fully visible or slightly occluded, thesecond mode is selected if there is a short-time critical occlusion inthe area, the third mode is selected if there is a longer lastingcritical occlusion in the area, and the last mode is selected if thereis a longer lasting critical occlusion in the area and a stable state ofone of the plurality of predetermined occupancy classes is not reached.

According to an embodiment, in the first mode 602 a change from anoccupancy class for the present point of time to a different occupancyclass of one of the plurality of predetermined occupancy classes may bebased on a time fused occupancy class prediction.

According to an embodiment, the selection of one of the plurality ofmodes may be provided from the first mode 602 into the second mode 604,if the occlusion value OCC for the present point of time is larger thana predetermined occlusion threshold TC, or from the second mode 604 intothe first mode 602, if the occlusion value OCC for the present point oftime is lower than the predetermined occlusion threshold TC; or fromsecond mode 604 into the third mode 606, if the second mode 604 has beenselected for a period of time longer than a predetermined blocking timethreshold TBT; or from the third mode 606 into the first mode 602, if atime fused occlusion value TOCC is lower than the predeterminedocclusion threshold TC; or from the third mode 606 into the last mode608, if a stable state of one of the plurality of occupancy classes isnot reached for a predetermined occlusion time threshold TOT, nooccupancy class reaches an increased confirmation threshold for thepredetermined occlusion time threshold TOT, or the occupancy class isconstantly changing for the predetermined occlusion time threshold TOT;or from the last mode 608 into the third mode 606, if a stable state ofone of the plurality of occupancy classes is reached or a user confirmsa change of the presently identified occupancy class; or from the lastmode 608 into the first mode 602, if the time fused occlusion value TOCCis lower than the predetermined occlusion threshold TC.

According to an embodiment, the predetermined occlusion threshold TC maydepend on the occupancy class for the present point of time.

According to an embodiment, the method further comprises the followingstep carried out by the computer hardware components: determining acondition of the vehicle, wherein selecting one of the plurality ofmodes is based on the condition of the vehicle.

According to an embodiment, the occlusion value OCC may be determinedbased on the measurement data and characteristics 204 related to arelevant region.

According to an embodiment, the method further comprises the followingstep carried out by the computer hardware components: determiningcharacteristics 204 related to a relevant region for the occupancy classfor the present point of time of the area from a plurality ofcharacteristics 204 for each possible predetermined occupancy class of aplurality of predetermined occupancy classes; and wherein the occlusionvalue OCC is determined based on the measurement data and the determinedcharacteristics 204 related to the relevant region.

According to an embodiment, the characteristics 204 related to therelevant region may depend on the presently identified occupancy classfor the present point of time and/or the characteristics 204 related tothe relevant region may include a weight mask 300, the weight mask 300may include a plurality of mask pixels, each mask pixel may correspondto a measurement pixel of the measurement data.

According to an embodiment, an occlusion prediction 502 may bedetermined based on the measurement data, a pixel segmentation and/or anobject classification.

According to an embodiment, the predetermined occupancy classes mayinclude at least two of the following: an empty seat E, an object O, achild seat C and a person P.

FIG. 8 shows a flow diagram 800 illustrating a method for occlusionvalue determination according to various embodiments. At 802,measurement data related to an area may be determined. At 804,characteristics 204 related to a relevant region for an occupancy classfor the present point of time may be determined. At 806, an occlusionvalue OCC based on the measurement data and the determinedcharacteristics 204 related to the relevant region may be determined.

Each of the steps 702, 704, 706, 708, 802, 804, 806, and the furthersteps described above may be performed by computer hardware components,for example as described with reference to FIG. 9 .

FIG. 9 shows a computer system 900 with a plurality of computer hardwarecomponents configured to carry out steps of a computer-implementedmethod for occupancy class prediction and/or of a computer-implementedmethod for occlusion value determination according to variousembodiments. The computer system 900 may include a processor 902, amemory 904, and a non-transitory data storage 906. A camera 908 and/or adistance sensor 910 (for example, a radar sensor or a LIDAR sensor) maybe provided as part of the computer system 900 (like illustrated in FIG.9 ), or may be provided external to the computer system 900.

The processor 902 may carry out instructions provided in the memory 904.The non-transitory data storage 906 may store a computer program,including the instructions that may be transferred to the memory 904 andthen executed by the processor 902. The camera 908 and/or the distancesensor 910 may be used to determine measurement data, for examplemeasurement data that is provided to the methods as described herein.

The processor 902, the memory 904, and the non-transitory data storage906 may be coupled with each other, e.g., via an electrical connection912, such as e.g., a cable or a computer bus or via any other suitableelectrical connection to exchange electrical signals. The camera 908and/or the distance sensor 910 may be coupled to the computer system900, for example via an external interface, or may be provided as partsof the computer system (in other words: internal to the computer system,for example coupled via the electrical connection 912).

The terms “coupling” or “connection” are intended to include a direct“coupling” (for example, via a physical link) or direct “connection” aswell as an indirect “coupling” or indirect “connection” (for example,via a logical link), respectively.

It will be understood that what has been described for one of themethods above may analogously hold true for the computer system 900.

LIST OF REFERENCE CHARACTERS FOR THE ELEMENTS IN THE DRAWINGS

The following is a list of the certain items in the drawings, innumerical order. Items not listed in the list may nonetheless be part ofa given embodiment. For better legibility of the text, a given referencecharacter may be recited near some, but not all, recitations of thereferenced item in the text. The same reference number may be used withreference to different examples or different instances of a given item.

-   -   E empty seat    -   C child seat    -   O object    -   P person    -   100 characteristics related to a relevant region for        predetermined occupancy classes    -   102 relevant region of a child seat    -   104 relevant region of an empty seat    -   106 relevant region of an object    -   108 relevant region of a person    -   200 relevant region and corresponding characteristics related to        the relevant region    -   202 frame    -   204 characteristics    -   206 other parts of an area    -   300 weight masks for predetermined occupancy classes    -   302 region of intensity of a child seat    -   304 region of intensity of an empty seat    -   306 region of intensity of an object    -   308 region of intensity of a person    -   400 occlusion examples of a camera view on a given seat    -   402 occlusion based on a head of a driver    -   404 occlusion based on an arm of a driver    -   406 occlusion based on an arm of a co-driver    -   408 occlusion based on an object    -   410 occlusion based on an arm of a rear-seat passenger    -   412 occlusion based on body parts of a rear-seat passenger or by        an object    -   500 occlusion value determination for an example occlusion        prediction and different predetermined occupancy classes    -   502 occlusion prediction    -   504 characteristics related to a child seat    -   506 characteristics related to an empty seat    -   508 characteristics related to an object    -   510 characteristics related to a person    -   600 flow diagram illustrating the relationship between the modes        for an occupancy class prediction based on the occlusion value    -   602 first mode represented as a regular mode    -   604 second mode represented as a short-term occlusion mode    -   606 third mode represented as a long-term occlusion mode    -   608 last mode represented as a HMI occlusion mode    -   OCC occlusion value    -   TC predetermined occlusion threshold    -   TOCC time fused occlusion value    -   TB blocking time interval    -   TBT predetermined blocking time threshold    -   TOT predetermined occlusion time threshold    -   700 flow diagram illustrating a method for occupancy class        prediction according to various embodiments    -   702 step of determining measurement data related to an area    -   704 step of determining an occlusion value based on the        measurement data    -   706 step of determining a mode for occupancy class prediction        based on the occlusion value    -   708 step of determining one of a plurality of predetermined        occupancy classes of the area based on the mode    -   800 flow diagram illustrating a method for occlusion value        determination according to various embodiments    -   802 step of determining measurement data related to an area    -   804 step of determining characteristics related to a relevant        region for a presently identified occupancy class    -   806 step of determining an occlusion value based on the        measurement data and the determined characteristics related to        the relevant region    -   900 computer system according to various embodiments    -   902 processor    -   904 memory    -   906 non-transitory data storage    -   908 camera    -   910 distance sensor    -   912 connection

What is claimed is:
 1. A method comprising: for a plurality of points oftime, determining measurement data related to an area; for the pluralityof points of time, determining occlusion values based on the measurementdata; for a present point of time, selecting one of a plurality of modesfor occupancy class prediction based on one or more of (i) the occlusionvalues for at least one of the present point of time and a previouspoint of time and (ii) the plurality of modes for occupancy classprediction selected for the previous point of time; and for the presentpoint of time, determining one of a plurality of predetermined occupancyclasses of the area based on a selected mode for the present point oftime, the determining sufficient for occupancy class prediction; andwherein the plurality of modes comprises a first mode, a second mode, athird mode, and a last mode, wherein the first mode is selected if thearea is fully visible or slightly occluded, the second mode is selectedif there is a short-time critical occlusion in the area, the third modeis selected if there is a longer lasting critical occlusion in the area,and the last mode is selected if there is a longer lasting criticalocclusion in the area and a stable state of one of the plurality ofpredetermined occupancy classes is not reached.
 2. The method of claim1, wherein in the first mode, a change from an occupancy class for thepresent point of time to a different occupancy class of one of theplurality of predetermined occupancy classes is based on a time fusedoccupancy class prediction.
 3. The method of claim 2, wherein selectingone of the plurality of modes is provided: from the first mode into thesecond mode, if the occlusion value for the present point of time islarger than a predetermined occlusion threshold; from the second modeinto the first mode, if the occlusion value for the present point oftime is lower than the predetermined occlusion threshold; from secondmode into the third mode, if the second mode has been selected for aperiod of time longer than a predetermined blocking time threshold; fromthe third mode into the first mode, if a time fused occlusion value islower than the predetermined occlusion threshold; from the third modeinto the last mode, if a stable state of one of the plurality ofoccupancy classes is not reached for a predetermined occlusion timethreshold, no occupancy class reaches an increased confirmationthreshold for the predetermined occlusion time threshold, or theoccupancy class is constantly changing for the predetermined occlusiontime threshold; from the last mode into the third mode, if a stablestate of one of the plurality of occupancy classes is reached or a userconfirms a change of a presently identified occupancy class; or from thelast mode into the first mode, if the time fused occlusion value islower than the predetermined occlusion threshold.
 4. The method of claim3, wherein the predetermined occlusion threshold depends on theoccupancy class for the present point of time.
 5. The method of claim 1,further comprising: determining a condition of a vehicle; and whereinselecting one of the plurality of modes is based on the condition of thevehicle.
 6. The method of claim 1, wherein the occlusion value isdetermined based on the measurement data and characteristics related toa relevant region.
 7. The method of claim 1, further comprising:determining characteristics related to a relevant region for theoccupancy class for the present point of time of the area from aplurality of characteristics for each possible predetermined occupancyclass of a plurality of predetermined occupancy classes; and wherein theocclusion value is determined based on the measurement data and thedetermined characteristics related to the relevant region.
 8. The methodof claim 1, further comprising: for the plurality of points of time,determining characteristics related to a relevant region for anoccupancy class for a present point of time of the area from a pluralityof characteristics for each possible predetermined occupancy class of aplurality of predetermined occupancy classes; and for the present pointof time, determining an occlusion value based on the measurement datafor the present point of time and the determined characteristics for thepresent point of time related to the relevant region, the determiningsufficient for occlusion value determination.
 9. The method of claim 8,wherein the characteristics related to the relevant region depend on theoccupancy class for the present point of time; or wherein thecharacteristics related to the relevant region comprise a weight mask,the weight mask comprising a plurality of mask pixels, each mask pixelcorresponding to a measurement pixel of the measurement data.
 10. Themethod of claim 8, wherein an occlusion prediction is determined basedon at least one of the measurement data, a pixel segmentation, or anobject classification.
 11. The method of claim 1, wherein the one of aplurality of predetermined occupancy classes comprise at least two ofthe following: an empty seat, an object, a child seat, or a person. 12.A method comprising: for a plurality of points of time, determiningmeasurement data related to an area; for the plurality of points oftime, determining occlusion values based on the measurement data; for apresent point of time, selecting one of a plurality of modes foroccupancy class prediction based on one or more of (i) the occlusionvalues for at least one of the present point of time and a previouspoint of time and (ii) the plurality of modes for occupancy classprediction selected for the previous point of time; for the presentpoint of time, determining one of a plurality of predetermined occupancyclasses of the area based on a selected mode for the present point oftime, the determining sufficient for occupancy class prediction; and forthe plurality of points of time, determining characteristics related toa relevant region for an occupancy class for a present point of time ofthe area from a plurality of characteristics for each possiblepredetermined occupancy class of a plurality of predetermined occupancyclasses; and for the present point of time, determining an occlusionvalue based on the measurement data for the present point of time andthe determined characteristics for the present point of time related tothe relevant region, the determining sufficient for occlusion valuedetermination.
 13. The method of claim 12, wherein the characteristicsrelated to the relevant region depend on the occupancy class for thepresent point of time; or wherein the characteristics related to therelevant region comprise a weight mask, the weight mask comprising aplurality of mask pixels, each mask pixel corresponding to a measurementpixel of the measurement data.
 14. The method of claim 12, wherein anocclusion prediction is determined based on at least one of themeasurement data, a pixel segmentation, or an object classification. 15.The method of claim 12, wherein the plurality of modes comprises a firstmode, a second mode, a third mode, and a last mode, wherein the firstmode is selected if the area is fully visible or slightly occluded, thesecond mode is selected if there is a short-time critical occlusion inthe area, the third mode is selected if there is a longer lastingcritical occlusion in the area, and the last mode is selected if thereis a longer lasting critical occlusion in the area and a stable state ofone of the plurality of predetermined occupancy classes is not reached.16. The method of claim 15, wherein in the first mode, a change from anoccupancy class for the present point of time to a different occupancyclass of one of the plurality of predetermined occupancy classes isbased on a time fused occupancy class prediction.
 17. The method ofclaim 16, wherein selecting one of the plurality of modes is provided:from the first mode into the second mode, if the occlusion value for thepresent point of time is larger than a predetermined occlusionthreshold; from the second mode into the first mode, if the occlusionvalue for the present point of time is lower than the predeterminedocclusion threshold; from second mode into the third mode, if the secondmode has been selected for a period of time longer than a predeterminedblocking time threshold; from the third mode into the first mode, if atime fused occlusion value is lower than the predetermined occlusionthreshold; from the third mode into the last mode, if a stable state ofone of the plurality of occupancy classes is not reached for apredetermined occlusion time threshold, no occupancy class reaches anincreased confirmation threshold for the predetermined occlusion timethreshold, or the occupancy class is constantly changing for thepredetermined occlusion time threshold; from the last mode into thethird mode, if a stable state of one of the plurality of occupancyclasses is reached or a user confirms a change of a presently identifiedoccupancy class; or from the last mode into the first mode, if the timefused occlusion value is lower than the predetermined occlusionthreshold.
 18. The method of claim 17, wherein the predeterminedocclusion threshold depends on the occupancy class for the present pointof time.
 19. The method of claim 12, further comprising: determining acondition of a vehicle; and wherein selecting one of the plurality ofmodes is based on the condition of the vehicle.
 20. The method of claim12, wherein the occlusion value is determined based on the measurementdata and characteristics related to a relevant region.